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The following is a quotation of the first paragraph of 35 U.S.C. 112: 

The specification shall contain a written description of the invention, and of the manner and process of 
making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the 
art to which it pertains, or with which it is most nearly connected, to make and use the same and shall 
set forth the best mode contemplated by the inventor of carrying out his invention. 

Claim 49 is rejected under 35 U.S.C. 112, first paragraph, as based on a 
disclosure which is not enabling. The visual recognition of the objects on a picture is 
critical or essential to the practice of the invention, but not included in the claim(s) is not 
enabled by the disclosure. See In re Mayhew, 527 F.2d 1229, 188 USPQ 356 (CCPA 
1 976). The claim 49 must include the feature of visual recognition of objects in order to 
identify the selected component for annotation. 

The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 1 02 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

Claims 44, 48-49 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Rose et al. (Annotating Real-World Objects Using Augmented Reality) in view of 
Harrison et al (6,611,725). 

As per claim 44, Rose teaches the claimed "computerized method for annotating" 
comprising the steps of: 
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"capturing a digital image of a view having an element with a camera" (Rose, 
page 2, video camera captures the object on scene; "currently we use a video monitor 
to represent the user's view, ..." - lines 2-3); 

"selecting an element from the captured image" (Rose, page 10, last paragraph); 

"obtaining an identification of the element" (Rose, page 9, section 6 Model 
Annotations, 2 nd paragraph); 

"relating the identification to annotating data associated with said elements; 
superimposing said annotating data on said view" (Rose, page 9, section 6 Model 
Annotations) ;and 

"causing the data to be displayed on a display device" (Rose, page 10, figure 

6.1). 

It is noted that Rose does not teach elements displayed with its annotating 
information is selected "from the captured image" as claimed. However, Harrison 
teaches how to generate an annotation to a display 2D object (Harrison, column 7, lines 
35-58). Furthermore, the visual recognition of the components on a picture "on the 
basis of the location of the mobile phone and visual cues which can be identified in the 
captured image" is well known in the art (Majumdar, section 3.3 Matching Model Data 
with Image Data, pages 362-363). The Majumdar reference is an example shows that 
in most of the cases, visual recognition is a complex process, but in Applicant's 
disclosure, the visual recognition is just barely mentioned so it assumes that these 
techniques were well known in the art. The decision to use a 3D pointer for selecting a 
part of the 3D object as in the Rose reference or a 2D pointer in combine with other 3D 
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data structure for selecting a part of an object represented in the 2D image as in 
Harrison and Majumdar references are both well known in the art. The motivation for 
use a 2D image to select a part for annotation is to take advantage of 2D monitor 
display. It is Rose does not teach the camera belongs to "a mobile phone" as claimed. 
However, since the mobile phone has no involvement in the process of capturing an 
image, selecting an element for annotating, finding and displaying the annotating data 
on the display, it would have been obvious to capture images by a camera of a mobile 
phone or a mobile device (e.g., a laptop) is just a matter of design choice. 
RESPONSE TO APPLICANT'S ARGUMENTS : 

Applicant's arguments filed December 1 , 201 1 have been fully considered but 
they are not deemed to be persuasive. The new reference of Majumdar is cited to show 
that the visual recognition of objects on a 2D picture is well known and widely practice in 
the art. The arguments are moot due to new ground of the rejection. It is also noted 
that, since Applicant's Specification just barely mentioned, but does not provide any 
details of how an object is visual recognized in a 2D picture, any visual specification 
technique used in the claims is just commonly known as mere design choice from the 
Prior art. 



As per claim 48, Rose teaches the claimed "system for annotating" comprising: 
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"a mobile phone including a camera for capturing a digital image of a view" 
(Rose, page 2, video camera captures the object on scene; "currently we use a video 
monitor to represent the user's view, ..." - lines 2-3); 

"means for obtaining an identification of the element" (Rose, page 9, section 6 
Model Annotations, 2 nd paragraph); 

"means for relating the identification to annotating data associated with said 
elements; means for superimposing said annotating data on said view" (Rose, page 9, 
section 6 Model Annotations); and 

"a display for displaying data" (Rose, page 10, figure 6.1). 

It is noted that Rose does not teach the elements displayed with its annotating 
information is selected "from the captured view" as claimed. However, Harrison teaches 
how to generate an annotation to a display 2D object (Harrison, column 7, lines 35-58). 
Furthermore, the visual recognition of the components on a picture "on the basis of the 
location of the mobile phone and visual cues which can be identified in the captured 
image" is well known in the art (Majumdar, section 3.3 Matching Model Data with Image 
Data, pages 362-363). The Majumdar reference is an example shows that in most of 
the cases, visual recognition is a complex process, but in Applicant's disclosure, the 
visual recognition is just barely mentioned so it assumes that these techniques were 
well known in the art. The decision to use a 3D pointer for selecting a part of the 3D 
object as in the Rose reference or a 2D pointer in combine with other 3D data structure 
for selecting a part of an object represented in the 2D image as in Harrison and 
Majumdar references are both well known in the art. The motivation for use a 2D image 
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to select a part for annotation is to take advantage of 2D monitor display. It is Rose 
does not teach the camera belongs to "a mobile phone" as claimed. However, since the 
mobile phone has no involvement in the process of capturing an image, selecting an 
element for annotating, finding and displaying the annotating data on the display, it 
would have been obvious to capture images by a camera of a mobile phone or a mobile 
device (e.g., a laptop) is just a matter of design choice. 

As per claim 49, Rose teaches the claimed "computerized method for annotating" 
comprising the steps of: 

"capturing a digital image of a view having an element with a camera; displaying 
said view on a display" (Rose, page 2, video camera captures the object on scene; 
"currently we use a video monitor to represent the user's view, ..." - lines 2-3); 

"selecting an element to be annotated" (Rose, page 10, last paragraph); 
"relating the identification to annotating data associated with said elements; 
superimposing said annotating data on said view" (Rose, page 9, section 6 Model 
Annotations) ;and 

"super imposing said annotating data on said view" (Rose, page 10, figure 6.1). 

It is noted that Rose does not teach elements displayed with its annotating 
information is selected "by pointing on said element on said display" as claimed. 
However, Harrison teaches how to generate an annotation to a display 2D object by 
pointing on said element on said display (Harrison, column 7, lines 35-58). 
Furthermore, the visual recognition of the components on a picture is well known in the 
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art (Majumdar, section 3.3 Matching Model Data with Image Data, pages 362-363). The 
Majumdar reference is an example shows that in most of the cases, visual recognition is 
a complex process, but in Applicant's disclosure, the visual recognition is just barely 
mentioned so it assumes that these techniques were well known in the art. The 
decision to use a 3D pointer for selecting a part of the 3D object as in the Rose 
reference or a 2D pointer in combine with other 3D data structure for selecting a part of 
an object represented in the 2D image as in Harrison and Majumdar references are 
both well known in the art. The decision to use a 3D pointer for selecting a part of the 
3D object as in the Rose reference or a 2D pointer in combine with other 3D data 
structure for selecting a part of an object represented in the 2D image as in Ellenby 
reference are both well known in the art. The motivation for use a 2D image to select a 
part for annotation is to take advantage of 2D monitor display. 

Claims 44, 48-49, and 51 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Rose et al. (Annotating Real-World Objects Using Augmented 
Reality) in view of Ellenby et al (6,307,556). 

As per claim 44, Rose teaches the claimed "computerized method for annotating" 
comprising the steps of: 

"capturing a digital image of a view having an element with a camera" (Rose, 
page 2, video camera captures the object on scene; "currently we use a video monitor 
to represent the user's view, ..." - lines 2-3); 
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"selecting an element from the captured image" (Rose, page 10, last paragraph); 

"obtaining an identification of the element" (Rose, page 9, section 6 Model 
Annotations, 2 nd paragraph); 

"relating the identification to annotating data associated with said elements; 
superimposing said annotating data on said view" (Rose, page 9, section 6 Model 
Annotations) ;and 

"causing the data to be displayed on a display device" (Rose, page 10, figure 

6.1). 

It is noted that Rose does not teach elements displayed with its annotating 
information is selected "from the captured image" as claimed. However, Ellen by 
teaches how to generate an annotation to a display 2D object (Ellenby, column 7, lines 
47-61). Furthermore, the visual recognition of the components on a picture "on the 
basis of the location of the mobile phone and visual cues which can be identified in the 
captured image" is well known in the art (Majumdar, section 3.3 Matching Model Data 
with Image Data, pages 362-363). The Majumdar reference is an example shows that 
in most of the cases, visual recognition is a complex process, but in Applicant's 
disclosure, the visual recognition is just barely mentioned so it assumes that these 
techniques were well known in the art. The decision to use a 3D pointer for selecting a 
part of the 3D object as in the Rose reference or a 2D pointer in combine with other 3D 
data structure for selecting a part of an object represented in the 2D image as in Ellenby 
and Majumdar references are both well known in the art. The motivation for use a 2D 
image to select a part for annotation is to take advantage of 2D monitor display. It is 
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Rose does not teach the camera belongs to "a mobile phone" as claimed. However, 
since the mobile phone has no involvement in the process of capturing an image, 
selecting an element for annotating, finding and displaying the annotating data on the 
display, it would have been obvious to capture images by a camera of a mobile phone 
or a mobile device (e.g., a laptop) is just a matter of design choice. 

RESPONSE TO APPLICANT'S ARGUMENTS : 

Applicant's arguments filed December 1 , 201 1 have been fully considered but 
they are not deemed to be persuasive. The new reference of Majumdar is cited to show 
that the visual recognition of objects on a 2D picture is well known and widely practice in 
the art. The arguments are moot due to new ground of the rejection. It is also noted 
that, since Applicant's Specification just barely mentioned, but does not provide any 
details of how an object is visual recognized in a 2D picture, any visual specification 
technique used in the claims is just commonly known as mere design choice from the 
Prior art. 

As per claim 48, Rose teaches the claimed "system for annotating" comprising: 
"a mobile phone including a camera for capturing a digital image of a view" 

(Rose, page 2, video camera captures the object on scene; "currently we use a video 

monitor to represent the user's view, ..." - lines 2-3); 

"means for obtaining an identification of the element" (Rose, page 9, section 6 

Model Annotations, 2 nd paragraph); 
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"means for relating the identification to annotating data associated with said 
elements; means for superimposing said annotating data on said view" (Rose, page 9, 
section 6 Model Annotations); and 

"a display for displaying data" (Rose, page 10, figure 6.1). 

It is noted that Rose does not teach the elements displayed with its annotating 
information is selected "from the captured view" as claimed. However, Ellenby teaches 
how to generate an annotation to a display 2D object (Ellenby, column 7, lines 47-61 ). 
Furthermore, the visual recognition of the components on a picture "on the basis of the 
location of the mobile phone and visual cues which can be identified in the captured 
image" is well known in the art (Majumdar, section 3.3 Matching Model Data with Image 
Data, pages 362-363). The Majumdar reference is an example shows that in most of 
the cases, visual recognition is a complex process, but in Applicant's disclosure, the 
visual recognition is just barely mentioned so it assumes that these techniques were 
well known in the art. The decision to use a 3D pointer for selecting a part of the 3D 
object as in the Rose reference or a 2D pointer in combine with other 3D data structure 
for selecting a part of an object represented in the 2D image as in Ellenby and 
Majumdar references are both well known in the art. The motivation for use a 2D image 
to select a part for annotation is to take advantage of 2D monitor display. It is Rose 
does not teach the camera belongs to "a mobile phone" as claimed. However, since the 
mobile phone has no involvement in the process of capturing an image, selecting an 
element for annotating, finding and displaying the annotating data on the display, it 
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would have been obvious to capture images by a camera of a mobile phone or a mobile 
device (e.g., a laptop) is just a matter of design choice. 

As per claim 49, Rose teaches the claimed "computerized method for annotating" 
comprising the steps of: 

"capturing a digital image of a view having an element with a camera; displaying 
said view on a display" (Rose, page 2, video camera captures the object on scene; 
"currently we use a video monitor to represent the user's view, ..." - lines 2-3); 

"selecting an element to be annotated" (Rose, page 10, last paragraph); 
"relating the identification to annotating data associated with said elements; 
superimposing said annotating data on said view" (Rose, page 9, section 6 Model 
Annotations) ;and 

"super imposing said annotating data on said view" (Rose, page 10, figure 6.1). 

It is noted that Rose does not teach elements displayed with its annotating 
information is selected "by pointing on said element on said display" as claimed. 
However, Ellenby teaches how to generate an annotation to a display 2D object by 
pointing on said element on said display (Ellenby, column 7, lines 47-61). Furthermore, 
the visual recognition of the components on a picture is well known in the art 
(Majumdar, section 3.3 Matching Model Data with Image Data, pages 362-363). The 
Majumdar reference is an example shows that in most of the cases, visual recognition is 
a complex process, but in Applicant's disclosure, the visual recognition is just barely 
mentioned so it assumes that these techniques were well known in the art. The 
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decision to use a 3D pointer for selecting a part of the 3D object as in the Rose 
reference or a 2D pointer in combine with other 3D data structure for selecting a part of 
an object represented in the 2D image as in Ellenby and Majumdar references are both 
well known in the art. The motivation for use a 2D image to select a part for annotation 
is to take advantage of 2D monitor display. 

As per claim 51 , Rose teaches the claimed "computerized method for annotating" 
comprising the steps of: 

"capturing a digital image of a view having an element with a camera; displaying 
said view on a display" (Rose, page 2, video camera captures the object on scene; 
"currently we use a video monitor to represent the user's view, ..." - lines 2-3); 

"relating a selected element to annotating data associated with the element" 
(Rose, page 9, section 6 Model Annotations) ;and 

"super imposing said annotating data on said view" (Rose, page 10, figure 6.1). 

It is noted that Rose does not teach elements displayed with its annotating 
information is selected "using said positions, said shooting direction of said camera and 
visual cues" as claimed. However, Ellenby teaches how to generate an annotation to a 
display 2D object by "using said positions, said shooting direction of said camera and 
visual cues" (Ellenby, column 4, lines 36-38, column 6, lines 56-66; column 7, lines 26- 
46). Furthermore, the visual recognition of the components on a picture "on the basis of 
the location of the mobile phone and visual cues which can be identified in the captured 
image" is well known in the art (Majumdar, section 3.3 Matching Model Data with Image 
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Data, pages 362-363). The Majumdar reference is an example shows that in most of 
the cases, visual recognition is a complex process, but in Applicant's disclosure, the 
visual recognition is just barely mentioned so it assumes that these techniques were 
well known in the art. The decision to use a 3D pointer for selecting a part of the 3D 
object as in the Rose reference or a 2D pointer in combine with other 3D data structure 
for selecting a part of an object represented in the 2D image as in Ellenby and 
Majumdar references are both well known in the art. The motivation for use "said 
positions, said shooting direction of said camera and visual cues" because they are well 
known to accurately identify a selected element on the displayed scene (Ellenby, 
column 5, lines 35-47). 

Claim 50 is allowable. 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to PHU NGUYEN whose telephone number is (571)272- 
7645. The examiner can normally be reached on M-F/8AM-4:30PM. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Kee Tung can be reached on (571) 272 7794. The fax phone number for 
the organization where this application or proceeding is assigned is 571 -273-8300. 
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Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a 
USPTO Customer Service Representative or access to the automated information 
system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 



/ Phu K. Nguyen/ 

Primary Examiner, Art Unit 2628 



